Concurrent Hierarchical Reinforcement Learning for RoboCup Keepaway
نویسندگان
چکیده
RoboCup Keepaway, originated from the RoboCup soccer simulation 2D challenge, has been widely used as a machine learning benchmark. In this paper, we present a concurrent hierarchical reinforcement learning approach to RoboCup Keepaway. Following the idea of hierarchies of abstract machines (HAMs), we write a partial policy as a HAM from the perspective of a single keeper, run multiple instances of the HAM, and use reinforcement learning to learn the optimal completion of the resulting joint HAM. Furthermore, we apply the idea of exploiting the intrinsic internal transitions within the HAM structure for more efficient learning. Experimental results confirm that the concurrent HAM approaches outperform the state of the art significantly on the very complex RoboCup Keepaway domain.
منابع مشابه
Efficient Reinforcement Learning with Hierarchies of Machines by Leveraging Internal Transitions
In the context of hierarchical reinforcement learning, the idea of hierarchies of abstract machines (HAMs) is to write a partial policy as a set of hierarchical finite state machines with unspecified choice states, and use reinforcement learning to learn an optimal completion of this partial policy. Given a HAM with deep hierarchical structure, there often exist many internal transitions where ...
متن کاملSpeeding Up HAM Learning with Internal Transitions
In the context of hierarchical reinforcement learning, the idea of hierarchies of abstract machines (HAMs) is to write a partial policy as a set of hierarchical finite state machines with unspecified choice states, and use reinforcement learning to learn an optimal completion of this partial policy. Given a HAM with potentially deep hierarchical structure, there often exist many internal transi...
متن کاملArgumentation Accelerated Reinforcement Learning for RoboCup Keepaway-Takeaway
Multi-Agent Learning (MAL) is a complex problem, especially in real-time systems where both cooperative and competitive learning are involved. We study this problem in the RoboCup Soccer KeepawayTakeaway game and propose Argumentation Accelerated Reinforcement Learning (AARL) for this game. AARL incorporates heuristics, represented by arguments in Value-Based Argumentation, into Reinforcement L...
متن کاملHalf Field Offense in RoboCup Soccer: A Multiagent Reinforcement Learning Case Study
We present half field offense, a novel subtask of RoboCup simulated soccer, and pose it as a problem for reinforcement learning. In this task, an offense team attempts to outplay a defense team in order to shoot goals. Half field offense extends keepaway [11], a simpler subtask of RoboCup soccer in which one team must try to keep possession of the ball within a small rectangular region, and awa...
متن کاملArgumentation-Based Reinforcement Learning for RoboCup Soccer Keepaway
Reinforcement Learning (RL) suffers from several difficulties when applied to domains with no obvious goal state defined; this leads to inefficiency in RL algorithms. In this paper we consider a solution within the context of a widely-used testbed for RL, that of RoboCup Keepaway soccer. We introduce Argumentation-Based RL (ABRL), using methods from argumentation theory to integrate domain know...
متن کامل